Long-Term Identity-Aware Multi-Person Tracking for Surveillance Video Summarization

In multi-person tracking scenarios, gaining access to the identity of each tracked individual is crucial for many applications such as long-term surveillance video analysis. Therefore, we propose a long-term multi-person tracker which utilizes face recognition information to not only enhance tracking performance, but also assign identities to tracked people. As face recognition information is not available in many frames, the proposed tracker utilizes manifold learning techniques to propagate identity information to frames without face recognition information. Our tracker is formulated as a constrained quadratic optimization problem, which is solved with nonnegative matrix optimization techniques. Tracking experiments performed on challenging data sets, including a 116.25 hour complex indoor tracking data set, showed that our method is effective in tracking each individual. We further explored the utility of long-term identity-aware multi-person tracking output by performing video summarization experiments based on our tracking output. Results showed that the computed trajectories were sufficient to generate a reasonable visual diary (i.e. a summary of what a person did) for different people, thus potentially opening the door to summarization of hundreds or even thousands of hours of surveillance video.